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ABSTRACT 


Computerized  administration  of  the 
Armed  Services  Vocational  Aptitude 
Battery  (ASVAB)  potentially  offers 
several  benefits  over  the  current  paper- 
and-pencil  administration.  One  benefit 
is  the  possible  expansion  of  predictor 
tests  beyond  the  dominant  mathematical 
and  verbal  orientation  of  the  ASVAB. 

This  research  memorandum  first  examines 
the  comparability  of  measurement  for  the 
computerized  and  paper-and-pencil 
tests.  Second,  several  new  tests  that 
measure  psychomotor  skills  (hand-eye 
coordination  and  manual  dexterity)  are 
examined  for  their  ability  to  expand  the 
range  of  dimensions  currently  assessed 
by  the  ASVAB,  as  well  as  to  improve  the 
prediction  of  performance  in  mechanical 
maintenance  jobs. 
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EXECUTIVE  SUMMARY 


The  Department  of  Defense  (DOD)  Is  investigating  the  potential  of  a 
computerized  adaptive  testing  (CAT)  version  of  its  selection  battery, 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) .  However,  before 
implementing  CAT-ASVAB,  it  is  necessary  to  confirm  that  this  computer¬ 
ized  version  of  ASVAB  results  in  the  same  measurements  as  the  paper-and- 
pencil  (PP)  version.  The  computerized  administration  of  ASVAB  also 
allows  for  the  possible  assessment  of  many  new  constructs  that  could  not 
be  measured  using  the  PP  format.  A  psychomotor  construct  is  one  such 
dimension.  The  purpose  of  this  research  memorandum  is  to  address  each 
of  these  issues:  comparability  of  factor  structure  of  CAT-ASVAB  and 
PP-ASVAB;  uniqueness  of  psychomotor  tests  to  the  ASVAB  structure;  and 
ability  of  psychomotor  measures  to  improve  prediction  of  mechanical 
performance. 

The  sample  was  composed  of  Marine  Corps  automotive  and  helicopter 
mechanics  who  were  tested  as  part  of  the  Job  Performance  Measurement 
(JPM)  project.  Only  complete  data  cases  were  used  for  the  factor 
analyses,  which  resulted  in  a  sample  size  of  1,252.  The  analyses  of 
incremental  validity  were  based  on  775  automotive  mechanics  who  had 
complete  predictor  and  hands-on  performance  criterion  data.  Two  sources 
of  aptitude  information  were  available:  enlistment  PP  aptitude  scores 
and  current  aptitude  scores  obtained  from  administration  of  the 
CAT-ASVAB.  Five  measures  of  psychomotor  skills  were  concurrently 
administered  vfith  CAT-ASVAB.  Two  of  these  tests  assessed  general 
hand-eye  coordination  and  the  other  tests  measured  dexterity.  Hands-on 
performance  tests  served  as  the  performance  criterion  for  the  ground 
mechanics . 

RESULTS 

Factor  Structure  of  PP  and  CAT-ASVAB 

A  four-factor  solution  of  the  ASVAB  subtests  resulted  in  a  simple 
structure  for  separate  analyses  of  PP  and  CAT  scores.  Examining  the 
factor-pattern  matrix  showed  that  the  CAT  speed  factor  was  more 
mathematically  oriented  than  the  PP  speed  factor.  Several  subtest 
differences  also  were  noted  between  the  CAT  and  PP  administrations. 
However,  care  must  be  taken  in  interpreting  factor -structure  differences 
for  the  subtests  because  of  the  unknown  effect  of  intervening  training 
on  the  CAT  scores  (relative  to  PP  scores  at  time  of  enlistment)  and  the 
analysis  of  selected  samples. 

Essentially  equal  coramunality  estimates  were  found  for  the  CAT  and 
PP  subtests,  which  showed  little,  if  any,  improvement  in  reliability  was 
gained  from  using  computerized  adaptive  tests.  Factor  intercorrelations 
showed  that  the  CAT  factors  were  less  correlated  and,  therefore,  may 
possibly  provide  more  differential  validity  than  the  PP  version. 
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Similar  factor  analyses  were  conducted  with  the  psychomotor  tests 
included  to  determine  their  uniqueness  relative  to  the  ASVAB 
dimensions.  The  dexterity  tests  and  the  coordination  tests  defined 
distinct  factors.  The  same  results  were  achieved  whether  based  on 
enlistment  PP  scores  or  CAT -ASVAB  scores.  Therefore,  all  of  the 
psychomotor  measures  satisfied  the  initial  requirement  of  being  unique 
and  different  from  the  constructs  currently  assessed  by  the  ASVAB. 

Incremental  Validity  of  Psychomotor  Measures 

The  ten  ASVAB  subtests,  time  in  service,  and  its  square  were 
excellent  predictors  of  automotive  hands-on  performance.  The  multiple 
correlations,  corrected  for  range  restriction  and  shrinkage,  were  .68 
and  .74  for  enlistment  PP  scores  and  CAT  scores,  respectively.  These 
multiple  correlations  served  as  the  base  for  measuring  increments  in 
validity  achieved  by  the  psychomotor  tests.  To  limit  the  number  of 
significance  tests  and  thereby  not  overly  capitalize  on  chance, 
aggregated  measures  of  the  dexterity  and  coordination  dimensions,  not 
their  respective  subtests,  were  entered  into  the  regression.  Less  than 
one  validity  point  was  gained  by  combining  these  two  aggregated 
psychomotor  measures  compared  to  the  corrected  enlistment  validities. 

No  improvement  in  validity  was  made  relative  to  the  concurrent  CAT 
aptitude  subtests.  Thus,  despite  the  uniqueness  of  the  psychomotor 
measures  evidenced  by  the  results  of  the  factor  analyses,  the  tests  were 
not  able  to  enhance  the  prediction  of  the  mechanical  hands-on 
performance  tests. 

CONCLUSIONS 

Aptitude  measurements  made  by  CAT-ASVAB  may  differ  slightly  from 
the  current  PP- ASVAB.  These  differences  may  have  implications  for 
changing  the  historical  meaning  and,  consequently,  the  validity  of  the 
service  aptitude  composites.  However,  these  results  should  not  be 
overinterpreted.  Factor  analyses  of  selected  samples,  even  with 
corrections  for  range  restriction,  pose  difficult  issues  for 
interpreting  factoring  outcomes.  There  are  also  unknown  effects 
associated  with  comparing  the  same  samples  on  aptitude  measures 
administered  at  different  times  with  substantial  intervening  events  that 
affect  the  predictor  tests  (i.e.,  training).  This  study  clearly  was  not 
designed  to  properly  or  fully  address  this  comparability  of  measurement 
issue.  Similar  analyses  should  be  conducted  and  reported  based  on 
sufficiently  large  samples  of  military  applicants. 

The  coordination  tests  and  the  dexterity  tests  were  found  to 
represent  unique  and  distinct  factors  relative  to  the  constructs 
measured  by  the  ASVAB.  The  two  sets  of  tests  were  also  unique  relative 
to  each  other  and  did  not  define  a  common  psychomotor  factor  as  was 
anticipated.  From  the  perspective  of  expanding  the  predictor  space  of 
the  ASVAB,  either  set  of  measures  would  accomplish  this  goal.  Despite 

the  uniqueness  of  the  coordination  and  dexterity  tests,  neither  set  of 
tests  was  able  to  add  significantly  to  the  ASVAB' s  prediction  of 
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mechanical  hands-on  performance.  This  is  not  to  say  that  the 
coordination  and  dexterity  tests  would  not  significantly  improve 
prediction  for  other  specialties  in  which  these  psychomotor  dimensions 
relate  to  job  performance. 

ASVAB  was  shown  to  be  an  excellent  predictor  of  mechanical  job 
performance.  New  tests  measuring  aptitude  dimensions  different  from 
those  of  the  current  ASVAB  are  not  necessary  for  selection  purposes. 
Large  payoffs  potentially  exist  in  the  realm  of  improved  recruit 
assignments  for  job  classification,  although  this  study  did  not  find 
such  gains  for  automotive  mechanics.  To  satisfy  the  stringent 
requirements  for  inclusion  of  a  new  test  into  the  joint-service  battery, 
the  test  must  demonstrate  incremental  validity  across  all  services  as 
well  as  multiple  jobs  within  a  service.  A  new  test  sufficiently  unique 
to  produce  incremental  validity  is  likely  to  measure  a  specialized  trait 
and  thus  will  improve  prediction  in  only  a  narrow  range  of  jobs. 
Therefore,  the  likely  benefit  of  any  new  predictor  test  is  not  in  the 
form  of  supplementing  or  revising  a  joint-service  instrument,  but  rather 
in  terms  of  potentially  addressing  service- specific  needs  for  improving 
classification  for  selected  specialties. 
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INTRODUCTION 


The  Department  of  Defense  (DOD)  is  investigating  the  potential  of  a 
computerized  adaptive  testing  (CAT)  version  of  its  selection  battery, 
the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) .  CAT-ASVAB 
potentially  promises  more  precise  aptitude  measurement,  reduced  testing 
time,  better  test  security,  and  more  efficient  test  administration, 
scoring,  and  reporting.  However,  before  implementing  CAT-ASVAB,  it  is 
necessary  to  confirm  that  this  computerized  version  of  ASVAB  results  in 
the  same  measurements  as  the  paper-and-pencil  (PP)  version. 

The  computerized  administration  of  ASVAB  also  allows  for  the 
possible  assessment  of  many  new  constructs  that  could  not  be  measured  in 
PP  format  (e.g.,  psychomotor  skills,  reaction  time,  memory).  Some  have 
argued  that  the  ASVAB  is  limited  in  the  number  of  constructs  that  it 
effectively  measures  and  that  all  of  the  predictive  power  results  from 
the  measurement  of  general  cognitive  ability  [1]  .  Consideration  of  new 
constructs  that  are  unique  relative  to  the  existing  ASVAB  content  may 
have  potential  for  effectively  expanding  the  range  of  predictors  and 
thereby  potentially  improving  the  overall  military  selection  and 
classification  system.  A  psychomotor  construct  is  one  such  dimension 
that  is  not  currently  measured  by  the  ASVAB. 

The  purpose  of  this  study  is  to  compare  the  factor  structure  of 
CAT-ASVAB  to  PP-ASVAB.  Past  factor  analysis  of  the  ASVAB  has  shown  that 
the  ten  subtests  can  typically  be  reduced  to  four  correlated  factors: 
verbal,  mathematical,  technical,  and  speed  [2].  Factor  solutions  for 
both  CAT-ASVAB  and  PP-ASVAB  will  be  constrained  to  four  factors  and  the 
structure  will  be  compared  to  note  similarity.  Next,  several 
psychomotor  measures  are  included  in  the  factor  analysis  of  the  ASVAB  to 
determine  if  the  overall  factor  structure  can  be  expanded  or  if  such  new 
measures  are  highly  related  to  the  current  ASVAB  subtests.  Although 
unique  structure  is  necessary  for  new  tests  to  potentially  augment  the 
ASVAB,  this  alone  is  not  a  sufficient  condition.  Such  new  measures  must 
be  shown  to  have  Incremental  validity,  above  and  beyond  what  the  current 
ASVAB  is  able  to  achieve.  Therefore,  the  incremental  validity  will  be 
assessed  for  these  psychomotor  measures  to  determine  the  extent  to  which 
they  improve  the  prediction  of  military  job  performance. 

METHOD 

Aptitude  Measures 

The  ASVAB  is  composed  of  ten  subtests  that  the  Services  combine 
into  different  composites  to  classify  recruits  as  eligible  for 
particular  military  jobs.  The  ASVAB  subtests  are  as  follows: 
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General  Science  (GS)  Coding  Speed  (CS) 

Arithmetic  Reasoning  (AR)  Auto  and  Shop  Information  (AS) 

Work  Knowledge  (WK)  Mathematical  Knowledge  (MK) 

Paragraph  Comprehension  (PC)  Mechanical  Comprehension  (MC) 

Numerical  Operations  (NO)  Electronics  Information  (El) 

Aptitude  information  for  each  of  these  subtests  was  available  at 
two  levels:  aptitude  scores  used  to  enlist  in  the  Marine  Corps  based  on 
a  PP  administration  of  the  ASVAB  and  current  aptitude  scores  obtained 
from  an  administration  of  the  CAT-ASVAB.  These  two  sources  of  aptitude 
information  are  distinguished  throughout  this  paper  by  prefixing  each 
subtest  abbreviation  with  ENL  for  enlistment  scores  and  CAT  for 
computerized  scores . 

To  achieve  motivated  testing  conditions  for  CAT-ASVAB,  examinees 
were  promised  that  their  aptitude  scores  of  record  would  be  changed  to 
reflect  improvements  during  the  computerized  administration.  Such  an 
incentive  is  significant,  as  many  young  Marines  seek  to  transfer  to 
other  occupational  fields  or  to  officer  programs.  About  40  percent  of 
the  Marines  tested  satisfied  the  criteria  for  improving  their  scores  of 
record. 

Psychomotor  Tests 

Five  measures  of  various  psychomotor  skills  were  concurrently 
administered  with  CAT-ASVAB.  Two  of  these  tests  assessed  general 
hand-eye  coordination:  one-hand  tracking  and  two-hand  tracking.  These 
two  tests  are  part  of  the  joint-service  Enhanced  Computer  Administered 
Testing  (ECAT)  project.  The  tests  required  a  special  response  pedestal 
with  a  joy  stick  for  one-hand  tracking  and  two  sliding  knobs  (one  moving 
vertically,  the  other  horizontally)  for  two-hand  tracking.  For  both 
tests,  a  cursor  moved  in  a  random  pattern  across  the  screen  and  the 
object  was  to  use  the  joy  stick  or  slides  to  move  a  cross-hair  so  as  to 
minimize  the  distance  between  the  moving  cursor  and  the  cross-hair. 

Each  test  was  composed  of  24  trials.  The  two  tests  were  counterbalanced 
in  administration  across  examinees.  Because  of  the  controlled 
presentation  of  the  tests,  no  time  limits  were  necessary.  Scores  for 
the  tests  were  the  average  root  mean  square  error  distance  over  all 
trials,  standardized  to  a  mean  of  50  and  a  standard  deviation  of  10  in 
the  sample  of  examinees. 

The  five  subtests  from  the  psychomotor  composite  of  the  General 
Aptitude  Test  Battery  (GATB)  were  also  administered.  These  five 
subtests  were  combined  into  three  aptitude  scores  that  were  used  as  the 
unit  of  analysis  for  this  study.  The  three  aptitude  scores  and  the 
constructs  they  measure  were  as  follows: 
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o  Motor  coordination- -ability  to  make  precise  movements 
with  hands  and  fingers 

o  Finger  dexterity- -ability  to  move  fingers  and  manipulate 
small  objects  with  the  fingers  rapidly  and  accurately 

o  Manual  dexterity- -ability  to  move  the  hands  easily  and 
skillfully. 

All  three  aptitudes  were  composed  of  highly  speeded  subtests,  but 
each  subtest  score  was  based  on  repeated  trials.  Finger  dexterity  and 
manual  dexterity  required  special  apparatus  boards  for  administration, 
whereas  motor  coordination  was  a  PP  test.  Scores  for  each  aptitude  were 
determined  from  norm  tables  in  which  each  aptitude  has  a  mean  of  100  and 
a  standard  deviation  of  20  in  the  reference  population  [3] . 

Criterion 

The  incremental  validity  of  the  psychomotor  tests  (i.e.,  their 
ability  to  improve  the  prediction  of  mechanical  performance)  depends 
partly  on  the  accuracy  of  the  criterion  as  a  measure  of  actual  job 
performance.  The  Marine  Corps  Job  Performance  Measurement  (JPM)  project 
has  produced  representative  hands-on  tests  that  require  performance  of 
job  tasks  under  standardized  conditions.  Thus,  the  tests  achieved  high 
fidelity  to  actual  job  conditions  and  assessed  the  same  skills  and 
abilities  required  to  perform  the  tasks  on  the  job.  These  performance 
tests  were  constructed  on  the  basis  of  extensive  job  analyses  and  a 
review  of  Marine  Corps  technical  manuals  and  training  materials.  Based 
on  considerable  input  and  review  by  Marine  Corps  subject-matter  experts 
and  job  incumbents,  tasks  were  sampled  for  testing  so  as  to  be 
representative  of  the  broad  and  diverse  requirements  of  mechanical 
jobs.  Extensive  tryouts  were  conducted  of  each  hands-on  task  to  ensure 
that  the  performance  test  accurately  assessed  the  skills  and  abilities 
required  to  perform  the  job.  Eighteen  of  the  20  hands-on  performance 
tasks  to  varying  degrees  involved  using  tools,  maneuvering  hands  in 
small  (often  unseen)  places,  removing  and  replacing  small  parts,  and 
setting  up  delicate  precision  measuring  equipment.  Therefore,  the 
hands-on  performance  test  contained  a  broad  range  of  psychomotor  skills 
that  included  coordination  and  dexterity. 

Sample 

The  sample  was  composed  of  Marine  Corps  automotive  and  helicopter 
mechanics  who  were  tested  as  part  of  the’ JPM  project.  Mechanics  were 
randomly  selected  for  testing  to  satisfy  pay- grade,  time -in- service,  and 
aptitude  requirements.  Each  Marine  was  tested  for  two  days;  one  day  was 
devoted  to  hands-on  performance  testing  and  the  second  day  covered 
written  performance  measures,  new  predictor  tests,  and  CAT-ASVAB.  Only 
complete  data  cases  were  used  for  the  factor  analyses,  which  resulted  in 
a  sample  size  of  1,252.  The  analyses  of  incremental  validity  were  based 
on  775  automotive  mechanics  who  had  complete  predictor  and  criterion 
data. 
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Factor  Analysis 

Correlation  matrices  of  all  ASVAB  subtests  and  the  five  psychomotor 
measures  were  computed  based  on  complete  data  for  both  the  enlistment 
and  CAT  aptitude  scores.  Because  the  number  of  women  in  the  original 
testing  sample  was  small,  and  because  there  is  evidence  that  the 
CAT-ASVAB  tends  to  underestimate  their  aptitude  in  Auto -Shop 
Information,  women  were  excluded  from  all  analyses. 

The  mechanics  who  participated  in  the  study  are  a  selected 
sample.  To  account  for  these  selection  effects,  the  sample  correlations 
were  corrected  for  range  restriction  by  using  a  multivariate  procedure 
based  on  all  ten  ASVAB  subtests  [4] .  The  1980  youth  population 
restricted  to  males  served  as  the  reference  population  from  which  all 
corrections  were  derived  [5]  .  All  factor  analyses  were  computed  on  the 
corrected  correlation  matrices. 

Based  on  historical  factor  analyses  of  the  ASVAB  [2],  the  factor 
solutions  were  constrained  to  four  factors.  An  iterative  principal 
factors  procedure  was  used  with  a  promax  rotation.  Squared  multiple 
correlations  were  used  as  the  initial  estimates  of  communalities .  Based 
on  factor  loadings  greater  than  0.30,  each  factor  was  named.  Separate 
analyses  were  conducted  for  both  the  enlistment  and  CAT  scores.  In 
addition  to  comparing  factor  loadings  and  the  variance  explained  by  each 
factor,  estimates  of  communality  and  interfactor  correlations  were 
noted. 

Similar  factor  analyses  and  comparisons  were  conducted,  including 
the  five  psychomotor  measures.  Multiple  analyses  were  conducted  with 
the  solutions  constrained  to  four  to  seven  factors.  The  simplest 
structure  was  selected  for  reporting. 

Incremental  Validity  Analysis 

Increments  in  validity  were  computed  simply  as  the  difference 
between  two  validity  coefficients  (increments  in  sums  of  squares  are 
also  presented).  The  multiple  correlation  between  all  ASVAB  subtests, 
time  in  service,  and  its  square,  with  the  hands-on  performance  test  for 
the  automotive  mechanics  was  computed  to  provide  the  basis  for  judging 
increments  in  validity  by  the  psychomotor  measures.  Time  in  service  was 
required  in  the  regression  because  of  the  significant  amounts  of  service 
by  most  personnel  (mean  -  38.7  months,  standard  deviation  -  22.4  months, 
range  -  4  to  168  months). 

To  reduce  capitalization  on  chance  caused  by  the  number  of 
significance  tests  to  be  made,  the  five  psychomotor  tests  were 
aggregated  and  then  entered  as  two  separate  variables  into  the 
regression.  Consistent  with  the  results  of  the  factor  analysis,  a 
coordination  variable  was  computed  by  standardizing  and  summing  the 
scores  for  the  one-hand  and  two-hand  tests.  Similarly,  the  three 
aptitude  scores  from  the  GATB  were  aggregated  according  to  documented 
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procedures  in  computing  the  psychomotor  composite  [3] .  If  these 
aggregated  scores  were  found  to  make  significant  improvements  above  the 
ASVAB  validity  base,  separate  significance  tests  would  then  be  conducted 
for  each  psychomotor  test. 

All  comparisons  of  multiple  correlations  were  based  on  estimates 
adjusted  for  the  number  of  predictors  to  correct  for  shrinkage. 

RESULTS 

Factor  Structure  of  PP-  and  CAT-ASVAB 

The  range -corrected  correlation  matrix  for  the  enlistment  and  CAT 
subtests  are  presented  in  table  1.  Note  the  increase  in  the  AS  subtest 
mean  score  and  substantial  reduction  in  standard  deviation  from  the 
enlistment  administration  to  the  more  current  assessment  by  the  CAT 
subtest.  Also,  the  correlations  among  the  subtests  were  similar  for  the 
enlistment  and  CAT  administrations,  except  for  the  AS  subtest.  The 
correlations  between  the  CAT  subtests  and  AS  subtest  were  consistently 
lower  than  the  same  correlations  based  on  the  enlistment  PP  subtests. 
Such  changes  and  differences  in  the  descriptive  statistics  may  reflect 
the  substantial  training  and  on-the-job  experience  received  by  the 
mechanics  since  their  enlistment. 

A  four- factor  solution  of  the  ASVAB  subtests  resulted  in  a  simple 
structure  for  separate  analyses  of  PP  and  CAT  scores.  (See  table  2.) 

The  variance  explained  by  each  factor  showed  that  the  variance  among  PP 
subtests  was  predominantly  explained  by  the  first  two  common  factors  of 
technical  skill  and  speed.  About  an  equal  percentage  of  the  variance  in 
CAT  subtests  was  explained  by  the  speed  and  verbal  factors.  Thus,  the 
CAT  version  of  ASVAB  has  become  slightly  more  verbal  than  technical  and 
continues  to  emphasize  the  speed  factor. 

The  factor  pattern  matrix  showed  that  the  CAT  speed  factor  was  more 
related  to  mathematics  than  the  PP  speed  factor.  The  CAT  El  subtest 
equally  stressed  verbal  and  technical  skills;  PP  El  loaded  mainly  on  the 
technical  factor.  The  CAT  MC  subtest  had  equal  loadings  on  the 
technical  and  math  factors;  the  PP  MC  subtest  was  more  technical.  PP  GS 
incorporated  the  verbal,  technical,  and  math  factors;  CAT  GS  loaded 
entirely  on  the  verbal  factor.  Similar  conclusions  were  found  if  both 
sources  of  aptitude  information  were  factor  analyzed  as  a  single  data 
set  (i.e.,  the  number  of  variables  equaled  20  as  opposed  to  two  sets  of 
10).  (See  table  3.)  Care  must  be  taken  not  to  overinterpret  these 
results  because  of  the  unknown  effects  of  training  on  the  CAT  scores  and 
because  selected  samples  were  analyzed. 
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Table  2.  Factor  pattern  matrix  for  ASVAB  subtests 


PP 

■ASVAB 

CAT' 

-ASVAB 

Sub test 
(loading) 

Variance 

explained 

Subtest 

(loading) 

Variance 

explained 

Verbal 

WK  (.83) 

1.56 

WK  (.98) 

2.42 

PC  (.54) 

GS  (.71) 

GS  (.45) 

PC  (.57) 

El  (.27) 

El  (.47) 

Math 

MK  (.85) 

1.68 

AR  (.65) 

1.07 

AR  (.63) 

MK  (.56) 

MC  (.25) 

MC  (.45) 

GS  (.25) 

PC  (.27) 

Tech 

AS  (.92) 

2.72 

AS  (.90) 

1.87 

MC  (.74) 

MC  (.45) 

El  (.61) 

El  (.40) 

GS  (.33) 

Speed 

NO  (.81) 

2.03 

NO  (.97) 

2.50 

CS  (.69) 

CS  (.53) 

MK  (.37) 

AR  (.25) 

Total  variance 

7.99 

7.86 

explained 

NOTE:  Analysis  based  on  correlation  matrices  corrected 
for  range  restriction.  Only  subtests  for  which 
loadings  exceeded  .25  are  listed. 


Table  3.  Factor  analysis  of  CAT-ASVAB  and  PP-ASVAB 
subtests  combined 


Factor  pattern  matrix 

Tech  Speed  Verb  Math 

Communal ity 
estimate 

ENLGS 

0.57 

0. 

81 

CATGS 

0.72 

0. 

87 

ENLAR 

0.59 

0. 

84 

CATAR 

0.73 

0. 

85 

ENLWK 

0.85 

0. 

90 

CATWK 

0.89 

-- 

0. 

89 

ENLPC 

0.65 

0. 

74 

CATPC 

0.57 

0.30 

0. 

71 

ENLNO 

0.78 

0. 

75 

CATNO 

0.62 

0.29 

0. 

73 

ENLCS 

0.84 

0. 

70 

CATCS 

0.62 

0. 

59 

ENLAS 

0.86 

0. 

83 

CATAS 

0.91 

0. 

69 

ENLMK 

0.67 

0. 

82 

CATMK 

0.72 

0. 

84 

ENLMC 

0.60 

0. 

74 

CATMC 

0.61 

-  - 

0.38 

0. 

77 

ENLEI 

0.48 

0.39 

0. 

78 

CATEI 

0.44 

-  - 

0.41 

-  - 

0. 

67 

Variance 

explained 

4.36 

4.20 

4.09 

2.86 

Interfactor  correlations 


Tech 

Speed 

Verb 

Math 

Tech 

1.00 

0.39 

0.65 

0.49 

Speed 

0.39 

1.00 

0.62 

0.64 

Verb 

0.65 

0.62 

1.00 

0.69 

Math 

0.49 

0.64 

0.69 

1.00 

NOTE: 

Only  loadings  greater  than  .25 
noted.  Correlation  matrix  was 
for  range  restriction. 

are 

corrected 
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As  shown  in  table  4,  essentially  equal  communality  estimates  were 
found  for  the  separate  factor  analyses  CAT  and  PP  subtests,  which 
indicates  that  little,  if  any,  improvement  in  reliability  was  gained 
from  the  computerized  adaptive  tests.  Tables  2  and  4  report  results 
from  the  same  analyses.  Substantial  improvement  was  noted  for  the  NO 
subtest,  which  is  more  accurately  controlled  and  administered  using  a 
computer.  However,  the  low  communality  observed  for  CS  (in  combination 
with  the  high  communality  noted  for  NO)  could  not  be  explained. 


Table  4.  Communality  estimates  for 
ASVAB  subtests 


Sub test 

Enlistment 

aptitude 

CAT 

aptitude 

GS 

.81 

.83 

AR 

.83 

.85 

WK 

.94 

.92 

PC 

.73 

.74 

NO 

.80 

.95 

CS 

.64 

.53 

AS 

.79 

.78 

MK 

.87 

.80 

MC 

.78 

.77 

El 

.79 

.69 

NOTE:  Analysis  based  on 

correlation 

matrices  corrected 

for  range. 

The  factor  intercorrelations  noted  in  table  5  showed  that  the  CAT 
factors  were  less  correlated  and,  therefore,  may  possibly  provide  more 
differential  validity  than  the  PP  version.  The  most  striking  difference 
in  factor  correlations  was  between  the  speed  and  technical  factors,  .19 
for  CAT  and  .51  for  PP. 


Table  5.  Factor  intercorrelations  for  ASVAB  subtests 


Enlistment  aptitude  CAT  aptitude 


Verb 

Math 

Tech 

Speed 

Verb 

Math 

Tech 

Speed 

Verb 

1.00 

1.00 

Math 

.69 

1.00 

.71 

1.00 

Tech 

.70 

.61 

1.00 

.57 

.46 

1.00 

Speed 

.62 

.67 

.51 

1.00 

.57 

.58 

.19 

1.00 

NOTE: 

Analysis 

based 

on  correlation 

matrices 

corrected  for 

range 

restriction. 
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Consideration  of  Fsychomotor  Measures 

The  correlations  of  table  1  showed  one-  and  two-hand  tracking  to  be 
strongly  related  to  each  other  (r  -  .78),  but  only  slightly  related  to 
the  three  dexterity  measures  of  the  GATB.  Among  the  dexterity  measures, 
the  finger  dexterity  was  least  related  to  motor  coordination  (r  «  .27); 
moderate  correlations  were  noted  among  the  GATB  tests.  The  two  ECAT 
tests  were  more  related  to  the  ASVAB  subtests  than  were  the  GATB 
tests;  both  ECAT  tests  were  about  equally  correlated  to  each  of  the 
ASVAB  subtests;  and  the  GATB  tests  correlated  more  with  the  two  speed 
subtests  of  the  ASVAB.  Again,  the  AS  subtest  had  markedly  smaller 
correlations  with  the  psychomotor  measures  than  did  any  other  ASVAB 
subtest. 

The  five  psychomotor  measures  were  added  to  ASVAB,  and  similar 
factor  analyses  were  conducted  to  determine  their  uniqueness  relative  to 
the  other  ASVAB  dimensions.  Successive  analyses  were  performed,  with 
the  number  of  factors  ranging  from  four  to  seven.  For  each  of  the 
analyses,  the  dexterity  tests  of  the  GATB  and  the  coordination  tests  of 
the  ECAT  defined  distinct  and  separate  factors;  never  did  these  tests 
load  highly  on  the  same  dimension.  Each  of  the  factor  outcomes  were 
examined  for  interpretability  and  simple  structure  and  the  six-factor 
solution  was  determined  to  be  the  best.  Tables  6  and  7  present  this 
outcome  respectively  for  enlistment  and  CAT  subtests. 

The  same  results  were  achieved  regardless  of  the  source  of  aptitude 
information.  That  is,  for  both  enlistment  and  CAT  scores,  the 
coordination  and  dexterity  factors  were  unique  and  distinct  dimensions, 
defined  by  exactly  the  same  psychomotor  measures,  and  had  no  ASVAB 
subtest  to  load  on  either  of  them.  The  motor  coordination  test  of  the 
GATB  also  had  a  significant  loading  on  the  speed  factor  that  is 
consistent  with  the  highly  speeded  nature  of  this  test.  The  correlation 
of  the  coordination  and  dexterity  factors  was  only  .32  and  .38,  based  on 
enlistment  and  CAT  aptitude  scores,  respectively.  The  communality  for 
both  motor  coordination  and  finger  dexterity  were  exceptionally  low 
relative  to  the  enlistment  and  CAT  subtests  and  the  other  psychomotor 
measures.  As  discussed  earlier,  the  GATB  psychomotor  tests  are  highly 
speeded,  yet  only  motor  coordination  loaded  on  the  speed  factor.  Such 
low  communalities  may  partially  result  from  the  low  reliabilities  of 
these  tests. 

Incremental  Validity  of  Fsychomotor  Measures 

Table  8  presents  the  correlations  between  the  ASVAB,  ECAT,  and  GATB 
subtests  with  the  mechanical  hands-on  performance  test.  All  correla¬ 
tions  between  the  hands-on  criterion  and  the  ASVAB  subtests  (both 
enlistment  and  CAT)  exceeded  those  correlations  between  hands-on  perfor¬ 
mance  and  the  GATB  and  ECAT  subtests.  The  two  ECAT  subtests  were 
moderately  related  to  hands-on  performance.  The  aggregate  score  of  the  three 
GATB  aptitudes  correlated  only  slightly  with  hands-on  performance,  .24. 
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Table  6.  Factor  analysis  of  enlistment  ASVAB  subtests  and 
psychomotor  subtests 


Factor  pattern  matrix 

'ommunality 

estimate 

Tech 

Speed 

Coord 

Dext 

Math 

( 

Verb 

ENLGS 

0.62 

0.81 

ENLAR 

0.30 

-  - 

-  - 

-  - 

0.59 

0.83 

ENLWK 

0.59 

-  - 

-  . 

-  - 

0.54 

0.92 

ENLPC 

0.44 

-  - 

-  - 

0.38 

0.74 

ENLNO 

-  - 

0.80 

•  - 

-  - 

-  - 

0.80 

ENLCS 

-  - 

0.66 

-  - 

-  - 

-  - 

0.65 

ENLAS 

0.99 

-  - 

-  - 

-  - 

-  - 

0.81 

ENLMK 

-  - 

-  - 

-  - 

0.79 

0.87 

ENLMC 

0.74 

-  - 

-  - 

-  - 

0.78 

ENLEI 

0.79 

-  - 

-  - 

0.79 

MOTOR 

-  - 

0.31 

-  - 

0.35 

0.36 

MANUAL 

-  - 

-  - 

-  - 

0.99 

0.95 

FINGER 

-  - 

-  - 

0.47 

-  - 

0.31 

ONEHAND 

-  - 

-  - 

0.87 

-  - 

-  - 

0.78 

TWOHAND 

-  - 

-  " 

0.87 

-  - 

“  “ 

0.80 

Variance  4.26 

explained 

1.96 

1.78 

1.67 

1.15 

0.38 

Interfactor  correlations 


Tech 

Speed 

Coord 

Dext 

Math 

Verb 

Tech 

1.00 

0.53 

0.49 

0.29 

0.57 

0.34 

Speed 

0.53 

1.00 

0.48 

0.41 

0.65 

0.48 

Coord 

0.49 

0.48 

1.00 

0.32 

0.46 

0.25 

Dext 

0.29 

0.41 

0.32 

1.00 

0.25 

0.17 

Math 

0.57 

0.65 

0.46 

0.25 

1.00 

0.58 

Verb 

0.34 

0.48 

0.25 

0.17 

0.58 

1.00 

NOTE:  Only  loadings  greater  than  .30  are  noted. 

Correlation  matrix  was  corrected  for  range 
restriction.  "Coord"  represents  a  coordination 
factor.  "Dext"  represents  a  dexterity  factor. 
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Table  7.  Factor  analysis  of  CAT-ASVAB  subtests  and  psychomotor  subtests 


Factor  pattern  matrix 

'ommunality 

estimate 

Verb 

Speed 

Coord 

Dext 

Tech 

( 

Math 

CATGS 

0.83 

0 

84 

CATAR 

0.31 

-  - 

•  - 

-  - 

0.50 

0 

84 

CATWK 

1.02 

0 

90 

CATPC 

0.71 

-  - 

-  - 

-  - 

0 

74 

CATNO 

-  - 

0.91 

-  - 

0 

85 

CATCS 

-  - 

0.56 

-  - 

0 

59 

CATAS 

-  - 

-  - 

-  - 

-  - 

0.86 

0 

78 

CATMK 

0.41 

-  - 

0.45 

0 

80 

CATMC 

0.31 

-  - 

0.33 

0.33 

0 

81 

CATEI 

0.57 

-  - 

-  - 

-  - 

0.36 

0 

70 

MOTOR 

0.41 

0.35 

0 

44 

MANUAL 

-  - 

0.89 

0 

77 

FINGER 

-  - 

-  - 

0.57 

-  - 

0 

36 

ONEHAND 

-  - 

-  - 

0.89 

-  - 

0 

80 

TWOHAND 

-  “ 

0.86 

“  “ 

-  “ 

-  “ 

0 

77 

Variance  3.59 

explained 

2.26 

1.82 

1.54 

1.35 

0.41 

Inter- factor  correlations 


Verb 

Speed 

Coord 

Dext 

Tech 

Math 

Verb 

1.00 

0.60 

0.50 

0.36 

0.52 

0.57 

Speed 

0.60 

1.00 

0.43 

0.44 

0.05 

0.42 

Coord 

0.50 

0.43 

1.00 

0.38 

0.31 

0.46 

Dext 

0.36 

0.44 

0.38 

1.00 

0.21 

0.22 

Tech 

0.52 

0.05 

0.31 

0.21 

1.00 

0.42 

Math 

0.57 

0.42 

0.46 

0.22 

0.42 

1.00 

NOTE:  Only  loadings  greater  than  .30  are  noted. 

Correlation  matrix  was  corrected  for  range 
restriction.  "Coord"  represents  a  coordination 
factor.  "Dext"  represents  a  dexterity  factor. 


All  of  the  psychomotor  measures  administered  in  this  study 
satisfied  the  initial  requirement  of  being  unique  and  different  from  the 
constructs  currently  assessed  by  the  ASVAB. 
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Table  8.  Correlation  of  ASVAB,  ECAT,  and  GATB 
subtests  with  mechanical  hands-on  performance  test 


ASVAB  sub test  Enlistment 


CAT 


GS 

.52 

.54 

AR 

.52 

.54 

WK 

.49 

.49 

PC 

.46 

.52 

NO 

.38 

.34 

CS 

.37 

.40 

AS 

.61 

.67 

MK 

.45 

.41 

MC 

.60 

.66 

El 

.58 

.59 

GATB  aptitudes 


Motor 

.11 

Manual 

.14 

Finger 

.32 

Dexterity 

.26 

ECAT  subtests 

Onehand 

.30 

Twohand 

.34 

Coordination 

.34 

NOTE:  Correlations  are  corrected  for  range 
restriction. 


Table  9  shows  that  the  ten  ASVAB  subtests,  time  in  service,  and  its 
square  provide  excellent  prediction  of  automotive  hands-on  performance. 
The  corrected  multiple  correlations  are  .66  and  .74  for  enlistment  and 
CAT  scores,  respectively.  These  multiple  correlations  serve  as  the 
basis  for  comparison  against  which  all  increments  in  validity  by  the 
psychomotor  tests  must  be  achieved. 

To  limit  the  number  of  significance  tests,  and  thereby  not  overly 
capitalize  on  chance,  the  aggregated  measures  of  the  dexterity  dimension 
and  the  coordination  dimension  were  both  entered  into  the  regression. 

The  increment  in  sums  of  squares  was  noted  and  the  increase  in  the 
adjusted  multiple  correlation  was  computed.  Table  9  shows  that  less 
than  one  validity  point  was  gained  by  the  combination  of  these  two 
aggregated  psychomotor  measures  over  the  corrected  enlistment 
validities.  No  improvement  in  validity  was  made  relative  to  the 
concurrent  CAT  aptitude  subtests.  Increments  in  the  sample  validities 
by  the  two  psychomotor  measures  were  slightly  more  but  still  not 
substantial.  Thus,  despite  the  uniqueness  of  the  psychomotor  measures 
evidenced  by  the  results  of  the  factor  analyses,  the  tests  were  not  able 
to  enhance  the  prediction  of  the  mechanical  hands-on  performance  tests. 
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Table  9.  Incremental  validity  for  psychoraotor  measures  above  ASVAB  and 
time -in- service  base3 


Sample  values 

Corrected  values 

df 

SS 

adj  R 

df 

SS 

adj  R 

Total 

774 

50,750.05 

774 

71,579.93 

Enlistment  Aptitude 

ASVAB  +  TIS 

12 

12,761.53 

.490 

10 

31,774.20 

.661 

+  Dext  +  Coord 

2 

787.52 

.013 

2 

553.23 

.005 

+  Dext 

1 

354.48 

.006 

1 

349.99 

.003 

+  Coord 

1 

513.58 

.009 

1 

254.84 

.002 

CAT  Aptitude 

ASVAB  +  TIS 

12  19,188.41 

.607 

10 

39,472.67 

.739 

+  Dext  +  Coord 

2  140.77 

.001 

2 

95.83 

.000 

+  Dext 

1  98.88 

.001 

1 

89.22 

.000 

+  Coord 

1  54.13 

.000 

1 

12.02 

.000 

NOTE:  Multiple  correlations  are  adjusted  for 

the  number  of  predictors 

(adj  R). 

"TIS"  represents  time 

in  service  and 

its  square. 

"Dext"  represents  the  three  dexterity 

tests  of 

GATB,  and  " 

Coord" 

represents 

the  two  coordination 

tests 

of  EC AT. 

a.  Time  in  service  cannot  be  defined  for  the  population  and,  therefore, 

the  variables 

(TIS  and  its  square)  are  not 

included  in  the  regression 

based  on  the  z 

orrected  correlations. 

CONCLUSIONS 

Unique  opportunities  exist  with  the  possible  advent  of  computerized 
testing  for  military  accession  in  the  Department  of  Defense.  Before 
such  opportunities  can  be  effected,  however,  the  comparability  of 
aptitude  measurement  afforded  by  the  computerized  battery  relative  to 
the  current  paper-and-pencil  tests  must  be  established. 

Aptitude  measurements  made  by  CAT-ASVAB  may  differ  slightly  from 
the  current  PP-ASVAB.  These  differences  may  have  implications  for 
changing  the  historical  meaning  and,  thereby,  the  validity  of  the 
service  aptitude  composites.  However,  these  results  should  not  be 
overinterpreted.  Factor  analyses  of  selected  samples,  even  with 
corrections  for  range  restriction,  pose  difficult  issues  for 
interpreting  factoring  outcomes.  There  are  also  unknown  effects 
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associated  with  comparing  tha  same  samples  on  aptitude  measures 
administered  at  different  times  with  substantial  intervening  events  that 
affect  the  predictor  testa  (l.e.,  training).  Thla  study  clearly  was  not 
designed  to  properly  or  fully  addreas  this  comparablllty-of -measurement 
issue.  Similar  analyses  based  on  sufficiently  large  samples  of  military 
applicants  should  be  conducted  and  reported. 

The  coordination  tests  of  the  ECAT  and  the  dexterity  tests  of  the 
GATB  were  found  to  represent  unique  and  distinct  factors  relative  to  the 
constructs  measured  by  the  ASVAB.  The  two  sets  of  tests  were  also 
unique  relative  to  each  other  and  did  not  define  a  common  psychomotor 
factor  as  was  anticipated.  From  the  perspective  of  expanding  the 
predictor  space  of  the  ASVAB,  clearly  either  set  or  both  sets  of 
measures  would  accomplish  this  goal. 

Despite  the  uniqueness  of  the  coordination  and  dexterity  tests, 
neither  set  of  tests  was  able  to  add  significantly  to  the  ASVAB' s 
prediction  of  mechanical  hands-on  performance.  In  fact,  the  ASVAB  was 
shown  to  be  an  excellent  predictor  of  mechanical  job  performance. 

Others  have  argued  that  the  variance  in  performance  for  technical  jobs 
such  as  automotive  mechanics  Is  a  function  of  cognitive  abilities 
(measured  by  tests  such  as  the  ASVAB)  and  not  psychomotor  skills, 
although  auch  skills  are  related  to  performance  [6],  This  is  not  to  say 
that  the  coordination  and  dexterity  tests  would  not  significantly 
improve  prediction  for  other  specialties  in  which  these  psychomotor 
dimensions  are  relevant  to  job  performance. 

Given  the  substantial  training  and  on-the-job  experience  received 
by  moat  military  job  incumbenta,  using  a  concurrent  design  for  aptitude, 
new  predictor,  and  criterion  assessment  may  be  questioned.  Although 
such  a  design  attempts  to  control  for  several  extraneous  sources  of 
variance,  It  may  also  affect  the  estimation  of  validity  coefficients  for 
the  new  predictor  tests  by  not  completely  measuring  the  full  range  of 
scores  on  the  new  predictor  tests.  A  longitudinal  design  in  which  all 
predictor  measures  are  administered  at  the  time  of  accession  is 
certainly  superior,  but  such  studies  are  extremely  difficult  to 
implement  and  may  not  provide  information  In  time  to  affect  the 
decision-making  process  for  revising  the  ASVAB. 

New  tests  measuring  aptitude  dimensions  different  from  those  of  the 
current  ASVAB  are  not  necessary  for  selection  purposes.  Large  payoffs 
potentially  exist  in  the  realm  of  improved  recruit  assignments  for  job 
classification,  although  this  study  did  not  find  such  gains  for 
automotive  mechanics.  To  satisfy  the  stringent  requirements  for 
Inclusion  of  a  new  test  Into  the  joint-service  battery,  the  test  must 
demonstrate  incremental  validity  across  all  services  and  also  apply  to 
multiple  jobs  within  a  service.  A  new  test  sufficiently  unique  to 
produce  incremental  validity  is  likely  to  measure  a  specialized  trait 
and  thus  will  Improve  prediction  in  only  a  narrow  range  of  jobs. 


-15- 


Therefore,  the  likely  benefit  of  any  new  predictor  test  would  not  be  in 
the  form  of  supplementing  or  revising  a  joint-service  instrument,  but 
rather  in  terms  of  potentially  addressing  service -specific  needs  for 
improving  classification  for  selected  specialties. 
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